{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "import IPython\n",
    "\n",
    "import openpifpaf  # pylint: disable=unused-import"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Car Keypoints\n",
    "\n",
    " <div style=\"text-align: right\"> by <a href=\"https://scholar.google.com/citations?user=f-4YHeMAAAAJ&hl=en\">Lorenzo Bertoni</a> and <a href=\"https://www.linkedin.com/in/duncan-zauss/\">Duncan Zauss</a>, 13/04/2021 </div> <br /> \n",
    "\n",
    "\n",
    "\n",
    "This section describes the [OpenPifPaf](https://github.com/openpifpaf/openpifpaf) plugin for vehicles. The plugin uses the [ApolloCar3D Dataset](http://apolloscape.auto/car_instance.html). For more information, we suggest to check our latest [paper](https://arxiv.org/abs/2103.02440): <br /> \n",
    "\n",
    "> __OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association__<br />\n",
    "> _[Sven Kreiss](https://www.svenkreiss.com), [Lorenzo Bertoni](https://scholar.google.com/citations?user=f-4YHeMAAAAJ&hl=en), [Alexandre Alahi](https://scholar.google.com/citations?user=UIhXQ64AAAAJ&hl=en)_, 2021.\n",
    ">\n",
    "\n",
    "\n",
    "## Predict \n",
    "Prediction runs as standard openpifpaf predict command, but using the pretrained\n",
    "model on vehicles. The flag `--checkpoint=shufflenetv2k16-apollo-24` will cause\n",
    "that our 24 keypoint version of the Shufflenet 16 (AP 76.1%) will be automatically\n",
    "downloaded. As an example, we have tried on an image of the streets of Saint Peterbourg:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "python -m openpifpaf.predict images/peterbourg.jpg \\\n",
    "  --checkpoint=shufflenetv2k16-apollo-24 -o images \\\n",
    "  --instance-threshold 0.05 --seed-threshold 0.05 \\\n",
    "  --line-width 4 --font-size 0\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import IPython\n",
    "IPython.display.Image('images/peterbourg.jpg.predictions.jpeg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Image credit: [Photo](https://commons.wikimedia.org/wiki/File:Streets_of_Saint_Petersburg,_Russia.jpg) by [Ninaras](https://commons.wikimedia.org/wiki/User:Ninaras) which is licensed under [CC-BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The car poses with 66 and 24 keypoints can be visualized with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openpifpaf.plugins.apollocar3d import constants\n",
    "\n",
    "with openpifpaf.show.Canvas.blank(dpi=75, nomargin=True) as ax:\n",
    "    video_66 = constants.plot3d_red(ax, constants.CAR_POSE_66, constants.CAR_SKELETON_66).to_html5_video()\n",
    "IPython.display.HTML(video_66)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with openpifpaf.show.Canvas.blank(dpi=75, nomargin=True) as ax:\n",
    "    video_24 = constants.plot3d_red(ax, constants.CAR_POSE_24, constants.CAR_SKELETON_24).to_html5_video()\n",
    "IPython.display.HTML(video_24)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess Dataset\n",
    "\n",
    "The preprocessing step converts the annotations into the standard COCO format.\n",
    "It creates a version with all 66 keypoints and also creates a sparsified version\n",
    "with 24 keypoints (shown above).\n",
    "To get started, download the challenge train file which contains the train and\n",
    "validation split:\n",
    "\n",
    "```sh\n",
    "mkdir data-apollocar3d\n",
    "cd data-apollocar3d\n",
    "# wget https://ad-apolloscape.cdn.bcebos.com/3d_car_instance_sample.tar.gz\n",
    "tar -xvf 3d-car-understanding-train.tar.gz\n",
    "rm 3d-car-understanding-train.tar.gz\n",
    "cd ..\n",
    "```\n",
    "\n",
    "Then convert to COCO format:\n",
    "\n",
    "```sh\n",
    "pip install opencv-python\n",
    "python3 -m openpifpaf.plugins.apollocar3d.apollo_to_coco --split_images\n",
    "```\n",
    "\n",
    "This script will create annotations with 24 keypoints and 66 keypoints simultaneously.\n",
    "The argument `--split_images` copies the original images in the new folders according\n",
    "to the train val split, slowing down the process. No need to use it multiple times.\n",
    "\n",
    "\n",
    "## Train\n",
    "The default is training with 66 keypoints:\n",
    "\n",
    "```sh\n",
    "python3 -m openpifpaf.train --dataset apollo \\\n",
    "--basenet=shufflenetv2k16 --apollo-square-edge=769 \\\n",
    "--lr=0.00002 --momentum=0.95  --b-scale=5.0 \\\n",
    "--epochs=300 --lr-decay 160 260 --lr-decay-epochs=10  --weight-decay=1e-5 \\\n",
    "--weight-decay=1e-5  --val-interval 10 --loader-workers 16 --apollo-upsample 2 \\\n",
    "--apollo-bmin 2 --batch-size 8\n",
    "```\n",
    "\n",
    "```sh\n",
    "python3 -m torch.distributed.launch --nproc_per_node=2 \\\n",
    "  -m openpifpaf.train --ddp \\\n",
    "  --lr=0.0003 --momentum=0.95 --b-scale=10.0 --clip-grad-value=10.0 \\\n",
    "  --epochs=450 --lr-decay 430 440 --lr-decay-epochs=10 \\\n",
    "  --batch-size=8 --loader-workers=16 \\\n",
    "  --weight-decay=1e-5 \\\n",
    "  --dataset=apollo --apollo-upsample=2 --apollo-bmin=2 --apollo-square-edge=769 \\\n",
    "  --checkpoint=shufflenetv2k16 --lr-warm-up-start-epoch=250\n",
    "```\n",
    "\n",
    "For smaller memory GPUs use `--square-edge=513`.\n",
    "To train with 24kps, you need to use the `--apollo-use-24-kps` flag.\n",
    "\n",
    "\n",
    "\n",
    "## Evaluation\n",
    "To evaluate the pretrained model, use:\n",
    "\n",
    "```sh\n",
    "CUDA_VISIBLE_DEVICES=0 python3 -m openpifpaf.eval \\\n",
    "--dataset=apollo --loader-workers=8 \\\n",
    "--checkpoint shufflenetv2k16-apollo-66 \\\n",
    "--force-complete-pose --seed-threshold=0.2 \\\n",
    "--apollo-eval-long-edge=0\n",
    "```\n",
    "\n",
    "To evaluate your own model, provide a path to your checkpoint with \n",
    "the `--checkpoint` argument. To evaluate an Apollo checkpoint with 24 keypoints,\n",
    "provide the `--apollo-use-24-kps` flag.\n",
    "\n",
    "\n",
    "## Everything else\n",
    "All PifPaf options and commands still stand, check them in the other sections of the guide.\n",
    "If you are interested in training your own dataset, read the section on a {doc}`custom dataset <plugins_custom>`.\n"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "ea6946363a43e80d241452ab397f4c58bdd3d2517da174158e9c46ce6717422a"
  },
  "kernelspec": {
   "display_name": "Python 3.9.4 64-bit ('venv3': venv)",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": ""
  },
  "metadata": {
   "interpreter": {
    "hash": "ea6946363a43e80d241452ab397f4c58bdd3d2517da174158e9c46ce6717422a"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
